In order to realize the accurate recognition of manual alphabets in the sign language video, this paper presented a sign language recognition algorithm based on DI_CamShift (Depth Image CamShift) and SLVW (Sign Language Visual Word). First, it used Kinect to obtain the video and depth image information of sign language gestures. Second, it calculated spindle direction angle and mass center position of the depth images to adjust the search window for gesture tracking. Third, an Ostu algorithm based on depth integral image was applied to gesture segmentation, then the Scale Invariant Feature Transform (SIFT) features was extracted. Finally, it built the SLVW bag of words and used SVM for recognition. The best recognition rate of single manual alphabet can reach 99.67%, and the average recognition rate is 96.47%.